Zoomorphic computer user interface

ABSTRACT

A zoomorphic computer for controlling a computer system includes an animal or humanoid shaped shell having an attached transceiver for two way communication with a computer system. A position detecting unit determines position of the zoomorphic shell relative to the computer system, with change of position of the zoomorphic shell relative to the computer system changing state the zoomorphic shell or the computer system. The zoomorphic shell can have movable elements such as arms or tails, attached to the zoomorphic shell, and support a feedback unit that communicates with the computer system, modifying position of a movable element in response to computer system output.

FIELD OF THE INVENTION

The present invention relates to a zoomorphic computer user interface.More particularly, the present invention relates to a computer processorand display having animal, human, or object shaped input/output devicesfor visual, auditory, grasping, gripping, or touching mediated userinput.

BACKGROUND AND SUMMARY OF THE INVENTION

Computer input and output is predominantly supported by keyboards andmice for input, along with cathode ray tube (CRT) monitors, flat paneldisplays, and audio speakers for output. While such input/output devicesare both powerful and widely available, they do have limitations. Forexample, novice computer users are often uneasy in the presence of "hightech" input/output devices, while even skilled computer users may avoidtypes of computer interactions because of the perceived effort requiredto accomplish a task.

To overcome adverse perceptions and facilitate common computerinteractions such as logging on, providing passwords, or enablingcommunication applications, the present invention contemplates the useof small auxiliary computers separate and distinct from a user's primarycomputer. To enhance appeal, such "tangible" computers can beconstructed to simulate conventional objects having widespread userrecognition. For example, a small, handheld computer used to providepasswords to various applications can be shaped as a stylized key. Auser can rotate the key in front of a computer monitor, with internalrotation sensors in the key triggering a wireless sending of a logon andpassword sequence to "unlock" a computer or computer application.

One particularly useful form of tangible computer is configured to havean animal or human shape. This would include real or fantastic animals(horses or unicorns), doll shapes or action figures, or even imaginarymonsters or aliens. Such zoomorphic computers are readily accepted byyounger computer users, and easily recognizable by any computer user.Advantageously, a rich symbolic or metaphoric vocabulary is available toguide use of such zoomorphic tangible computers. For example, a growlingdog figure can be recognizable as a "guard dog", and may provide asuitable substitute to the stylized key tangible computer for passwordprotection of a main computer.

As those skilled in the art will appreciate, reliably, quickly, andintuitively transmitting complex commands to small tangible orzoomorphic computers can be difficult. Small computer devices do notgenerally have sufficient computer processing power to respond reliablyto voice or handwritten (pen based) commands, keyboards are often absentor too small for accurate finger input, and conventional buttons are toolarge or support too limited a command instruction set. User interfacetechniques that rely on bulky external modules (full size infraredlinked keyboards, tethered data gloves, or camera based gesturalrecognition equipment, for example) are expensive, often not readilyavailable outside selected sites, and probably too awkward forwidespread use in conjunction with consumer level portable computingdevices.

User interface designers for traditional, non-zoomorphic portablecomputers have attempted to compensate for some of these problems byconstructing devices that rely on various spatial, positional, orenvironmental cues that manually or automatically allow for activationof various modes in the device. For example, some laptop computers usethe action of opening/closing the lid to initiate automaticbootup/powerdown of the computer without requiring any additional signalinput (e.g. such as depressing a "start" button or typing "I₋₋ o₋₋ g₋₋o₋₋ n" on a keyboard) from a user. Alternatively, the use of smallportable computers that automatically switch control modes depending onposition, orientation, or room location have been investigated.Buttonless manual control of a portable computer through deliberate useractions such as tilting the portable computer have also been described.

However, all these solutions for interfacing with small portablecomputers have generally been limited in scope and functionality. Whatis needed is a user interface system suitable for even very smallportable computers (having volumetric dimensions on the order of a onecubic centimeter) that is powerful, can be intuitively operated by anordinary user with little training, and is still readily capable ofmodification or extension by the user. The present invention meets theserequirements by providing a manipulatory user interface that responds toa user twisting, folding, bending, squeezing, shaking, tilting,spinning, lifting, or otherwise physically manipulating the zoomorphiccomputer.

In the manipulatory user interface system of the present invention, themost basic level of manipulation is known as a "senseme". A senseme isdefined as a single indivisible type of physical manipulation. A partiallist of categories of sensemes include material transformations such assqueezing, twisting, stretching; local spatial transformations such astranslation, rotation, orbiting; and environmental transformations basedon temperature, light level, or vibration. For example, a small portablecomputer may support a deformable piece having multiple embedded sensorsthat detect folding, twisting, or bending of the deformable piece by auser. This computer can also contain a number of accelerometers thatsense relative spatial information; gyroscopic, radio or infraredpositional sensors for determining absolute position; and variousthermal or photosensors that respectively detect temperature and lightlevel changes. Intentional or unintentional modifications detected byone or more of these sensor systems can provide the basis for a powerfuluser interface scheme.

As will be appreciated, each senseme category contains many individuallydistinguishable members. For example, consider the category of sensemeknown as a "pinch", a structural transformation generally completed by auser squeezing the deformable piece between a forefinger and thumb. Apinch can be modified by varying its speed (quick or slow pinch),magnitude/intensity (light or hard pinch), portion of deformable piecepinched (top, bottom, or center of deformable piece pinched), or evenportion of body used to pinch (right handed pinch or left handed pinch),with each modification being distinguishable as a senseme capable ofbeing mapped onto a computer control command.

Although the wide variety of easily distinguishable sensemes would aloneprovide a powerful user interface to a computer, the present inventionfurther extends the flexibility of the senseme based user interface bysupporting computer control based on a "morpheme" input. The morpheme isa temporally synchronous (or overlapping asynchronous) tuple of one ormore sensemes. Note that a morpheme can (and often will) contain morethan one senseme. The sensemes combined into a morpheme can come eitherfrom the same category (the user pinches with a right hand while tappingwith a left hand finger), or different categories (the user pinches thedeformable piece with a right hand while modifying the spatial positionof the portable computer by tilting it forward).

Any morpheme can in turn be extended by participation in a "sentence. Asentence is defined as a sequence of one or more temporally disjointmorphemes. The sentence level allows definition of a physicalmanipulatory grammar by appropriate choice of morpheme sequence, andcorollary rules governing, for example, use of active (verb like)morphemes, naming (noun) morphemes, or connectors. Other possiblegrammar constructs used in sentences may include those based on "home"systems. Home systems are general-purpose gestural languages, whosegrammar and syntax are not borrowed in any way from a host language.Examples of these languages are gestural languages developed by deafchildren of hearing parents who have not been exposed to American SignLanguage (ASL), and the "plains talk" of North American Indians, whichwas used as a trade language.

Accordingly, the present invention provides a method for inputtinginformation to a computer connected to a deformable piece that can bemanipulated, and optionally to various position sensors (both relativeand absolute), pressure sensors, thermal sensors, or even light sensors.The method comprises the steps of manipulating the deformable piece toprovide a first morpheme input to the computer, with the first morphemeinput normally triggering a first default action by the computer. Thedeformable piece may also be manipulated to provide a second morphemeinput to the computer, with the second morpheme input converting thenormally triggered first default action to a second action. The firstand second morphemes (and any subsequent morphemes) together form asentence that can be interpreted as a command to implement a computercontrolled action, whether it be to unlock an electronically controlleddoor, display a graphical image on a computer display, or begin loggingon to a computer network. Advantageously, such a user interface systemis well suited for interaction with small computer devices, and may evenbe cross-cultural to a limited extent, with ability to squeeze or pinchbeing universal human attributes.

The present invention is particularly useful for portable computers thatcan be held in the palm of a hand. If the portable computer is partiallyor completely surrounded with a deformable material having embedded orcontact pressure/deformation sensors, a user is able to gesture with ormanipulate the whole computer in order to achieve some desired result.Material deformation can be implemented at various scales. For instance,a computer and connected deformation sensors mounted inside a flexibletube could respond to right angle bending of the tube, or even complexknotting or looping of the tube. In practice however, only minutesurface deformations are required, being just sufficient to providetactile feedback to pressing, pinching, or bending type manipulations.In either case, however, the measurement of location and pressureapplied to a surface is sufficient to characterize the mode ofinteraction (distinguishing a pinch from a prod).

One particularly preferred embodiment of a handheld portable computerthat responds to a physical manipulatory grammar in accordance with thepresent invention includes a computer, a feedback module to providevisual, auditory, or tactile feedback to a user (e.g., processorconnected LCD display, audio speaker, or tactile display to presentBraille or other conventional touch interface), and co-mountedgraspable, deformable piece partially or completely surrounding thefeedback module. In addition, various thermal or pressure sensors aremounted to detect handedness of a user, with the grasping hand generallybeing the non-dominant hand for the user. Depending on the handedness ofthe user (which can be considered as a morpheme), the displayed datastructure is modified. For example, text displayed on an LCD displayscreen may be automatically shifted rightward on the screen to allow penbased annotation on the left side of the screen, thereby aiding lefthanded users.

Physically manipulatable user interfaces additionally provide anopportunity for multiple devices to interact in a user friendly manner.For example, a tileable display system having multiple displays withtouch sensitive deformable pieces surrounding the display can be used toorganize documents based on relative position of contact of eachdisplay. For example, if two displays initially showing separate datastructures (e.g two different pages from two different electronic books)are brought together in side by side touching contact, the displayedvisual information can alter (e.g. by displaying adjacent pages of asingle electronic book). As will be appreciated, one can also usemultiple computers connected to physically manipulatable controlelements to construct complex commands for organizing data structures.

Additional functions, objects, advantages, and features of the presentinvention will become apparent from consideration of the followingdescription and drawings of preferred embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of zoomorphic computer in a shape of ateddy bear, the bear supporting visual and non-visual status displays,and a pressure sensor array for detecting surface deformations;

FIG. 2 is a graphical diagram illustrating various physical manipulationmorphemes, with axes respectively illustrating complexity of sensemetuples required to form a morpheme, and plasticity of a device requiredto support the physical manipulation;

FIGS. 3-16 schematically illustrate various preferred classes ofphysical manipulation morphemes;

FIG. 17 is a graphical diagram illustrating various spatial morphemes,with a first axis illustrating complexity of senseme tuples required toform a morpheme, and a second axis illustrating the degree of positionalinformation needed to support the physical manipulation (moving along acontinuum from relative local measurement along a single specifieddimension to absolute global measurement with six degrees of freedomdetermined);

FIGS. 18-26 schematically illustrate various preferred spatialmanipulation morphemes;

FIG. 27 is a graphical illustration showing increasing degrees of sensorsystem complexity that can be used to detect various categories ofenvironmental stimuli, including light effects, thermal effects, theelectromagnetic environment, and the vibratory/acoustic environment;

FIG. 28 is a graphical diagram illustrating various physicalmanipulation morphemes for multiple interacting devices, with axesrespectively illustrating the complexity of senseme tuples required toform a morpheme, and increasing level of physical contact;

FIGS. 29-34 schematically illustrate various preferred manipulationmorphemes for multiple interacting devices;

FIGS. 35 and 36 are schematic diagrams illustrating "squeeze" and "tilt"morphemes applicable to a portable computer;

FIG. 37 is a schematic diagram illustrating tilt and squeeze morphemesused to control view of large two dimensional data sets with arelatively small display of a portable computer;

FIG. 38 is a schematic illustration representing a portable computerhaving a display ready to receive annotations from a right handed user;

FIG. 39 is a schematic illustration representing a portable computerhaving a display ready to receive annotations from a left handed user;

FIG. 40 is an electronic schematic illustrating components of thepressure and tilt sensitive modules of a portable computer such asillustrated in FIGS. 35-39.

FIGS. 41 and 42 are schematic illustrations of a scanner/printer/copierusing a paper shaped display interface to support morphemic input;

FIG. 43 is a schematic view of a tangible computer interface formed as astylized key, with rotation of the key enabling logon to a passwordprotected personal computer;

FIG. 44 is a schematic view of a zoomorphic device enabling logon to apassword protected personal computer;

FIG. 45 is a schematic illustration of a zoomorphic device in the shapeof a penguin used as an audiovisual control system; and

FIG. 46 is a schematic illustration of the zoomorph of FIG. 45alternatively used in an electronic mail notification system;

DETAILED DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a teddy bear shaped zoomorphic embodiment of thepresent invention suitable for supporting a morphemic user interfacegrammar. Support of the grammar can require detection of a user'sphysical manipulation of a zoomorphic device, detection of relative orabsolute spatial location of the device, detection of variousenvironmental factors acting on the device, and even detection andinteraction with multiple devices or external computer networks. Asillustrated, a zoomorphic device 10 has a deformable surface 20 with anunderlying deformation sensor mesh 22 for detecting surface deformationacross or within multiple subregions of the deformable surface 20. Thedeformation sensor mesh 22 is connected to an internally containedprocessor 24 having associated memory system 26. For detecting variouspositional or environmental variables, a sensing system 28 is alsoprovided. The illustrated device further includes a feedback module 33,which may include an externally visible status display 30 or anon-visual feedback module 31 (typically delivering auditory or tactilefeedback). In the illustrated device, a communications system 32 forreception or transmission of information to other electronic orcomputing devices is also provided. All these components can be poweredby a power supply 25, which is usually an internally mountedrechargeable battery of conventional construction.

Whatever the shape of device 10, for operation of the present inventionthe device 10 is completely or partially enveloped by the deformableshell or surface 20. The present invention supports use of a greatvariety of designs and materials for the deformable surface 20,depending on the required plasticity, durability, longevity, and ofcourse, cost constraints. For example, contemplated designs fordeformable surface 20 include, but are not limited to:

plush cloth or textile material, optionally supported by foam orplastics, including closed or open celled polymeric foam material havinga wall thickness of millimeters to centimeters, with thinner walledembodiments being supported (e.g. by adhesive attachment) by an internalhard shell (constructed from polymeric or metallic materials), and thosethicker walled embodiments directly supporting (by, e.g. brackets orsupports) internal components such as processor 24. Suitable foams mayinclude those composed in whole or in part of widely available syntheticrubbers such as polychloroprene (neoprene), polystyrenes, rubber ornitrile rubber latex foams, polysiloxanes, block polymers includingstyrene-butadiene or styrene isoprene, or any other conventionalmaterial having good elasticity and deformability;

a thin single layer polymeric surface loosely wrapped around a internalhard shell (the hard shell being constructed from polymeric or metallicmaterials). For example, a nylon or cotton weave, single layerpolyethylene, synthetic rubber (with little or no foam cells present),or natural polymeric materials such as leather wrapped around apolystyrene casing can be used;

a composite layered surface having a durable polymeric outer layersupported by an inner foam layer; or even

a polymeric bilayer having an intermediate fluid or gel layer of aviscous or thixotropic material that can be used to support extremedeformations. The intermediate layer can be relatively thick (one theorder of centimeters), or in certain embodiments can have a thicknessmeasured on micron to millimeter scales. Such extremely thin layerswould allow complex twisting, folding, curling, or crumpling actions,and have been described in conjunction with U.S. Pat. No. 5,389,945,assigned to Xerox Corp., the disclosure of which is herein specificallyincorporated by reference.

The deformation sensor mesh 22 can be embedded within, or positioned tocontact, the deformable surface 20. The deformation sensor mesh 22 caninclude an array of individual compressional or tensional strainsensors, or alternatively, embedded or attached positional sensors. Forcertain applications, continuous sensors (e.g. bilayer sheets ofcapacitance sensors) may be employed. One particularly useful continuoussensor type uses multiple capacitance or resistance strips, withdeformation pressure resulting in a positionally localizable analogsignal proportional to the applied deformation pressure. Various sensortypes can be used, including simple capacitance sensors, resistivestrain sensors, analog or digital pressure switches, inductive sensors,or even fluid flow sensors. Depending on the sensor type employed,sensor data can be directly fed to the processor 24 in digital form, orbe transformed to digital format by an general purpose analog/digitalconverter that typically provides a 4 or 8 bit range (although as few asone or as many as 32 bits may be required by various applications). Ananalog to digital converter may be internal to the processor 24 orprovided as an external module. As will be appreciated, the sensor mesh22 is intended to include combinations of sensors and sensor types,which can be used over the whole or part of the deformable surface 20.

A positional or environmental sensor system 28 can also be supported byzoomorphic device 10. Various sensor modes can be supported, includingabsolute or relative positional information as determined by gyroscopicsensors, accelerometers, or acoustic or infrared ranging techniques.Environmental sensors, including conventional light, image, thermal,electromagnetic, vibratory, or acoustic sensors can also be present.Depending on the desired application, even costly environmental orpositional sensors such as those incorporating differential GPSpositioning, image analysis or recognition, acoustic or voiceidentification, or differential thermal sensors can be used as morphemicinput. Such morphemic input, taken in conjunction with morphemic inputas detected by sensor mesh 22, can enhance precision and flexibility ofa user's control of zoomorphic device 10.

As illustrated, both sensor system 28 and sensor mesh 22 are connectedto the processor 24 and associated memory 26. The processor 24 andmemory 26 are typically mounted within the deformable surface 20, byeither direct attachment to the deformable surface 20 or by attachmentto a hard casing positioned within the deformable surface 20.Conventional CISC or RISC processors can be used in the illustratedembodiment, with low power processors such as the Signetics 87c752 or87c751, Motorola 68HC11 or 68582, or ARM 710 being preferred. Ifconvenient, coprocessors such as analog to digital converters or digitalsignal processors can be used alone or in conjunction with a mainprocessor. Conventional flash, static, or dynamic RAM can used in thepresent invention, although for certain applications higher costembedded DRAM may also be used. In some storage intensive applications,memory 26 can include additional harddisk storage, either located withinthe device 10 or available through an external connection. As will beappreciated, for many applications use of optional externalcommunications can at least partially supplant use of internalprocessors and memory (except for that necessary to support requiredsensor or communication buffering and signalling).

The present invention optionally supports communications with anexternal computer system 40 using its internal communications system 32and associated transceiver 34. The external computer system 40 alsoincludes a transceiver 42, a personal computer or workstation 44, and isconnected to a local or wide area network computer system 46. Thetransceivers 34 and 42 can support various communication protocols anddesigns, including use of a serial tethered line 36 (using, for examplethe RS-232C interface protocols), use of infrared signals 38 adhering towidely utilized IRDA communication standards, or use of radiofrequencysignal 37 (which can be, for example, a cellular telephone, 900 MHzradio, or digital PCS telephonic communications). Alternativecommunication standards, or even alternative communication carriers suchas those based on optical or acoustic techniques, can of course beemployed.

As will be appreciated, in addition to direct communication withexternal computer system 40, the device 10 can be directly or indirectlymaintained in continuous or intermittent communication with a number ofsuitably equipped electronic devices, including a tablet computer 110,or even another zoomorphic portable computer 11 similar function todevice 10. Communication can be direct to a target device, or through anintermediary retransmitter such as computer system 40. Other possiblecommunication targets include automation control systems, securityauthorization units, personal digital assistants, notebook computers, orany other suitably equipped electronic system.

Results of communications with external devices, presentation of devicestored information, or device status updates can all be provided to auser through processor 24 controlled update of feedback module 33.Feedback to a user can be primarily visual, such as can occur inconjunction with visual display 30. Generally, the display 30 can be aconventional passive or active matrix liquid crystal display, althoughuse of more sophisticated (and expensive) displays based on variouselectrooptical or micromechanical techniques can of course be used. Inaddition, for certain devices a non-imaging display such as may beformed by a small number of status lights (e.g. red or green LEDs), orlocalized or distributed chromatic changes (in conjunction with adeformable surface 22 constructed with suitable electrochromicmaterials) may be all that is necessary for visual feedback to the user.

In some embodiments of the invention, visual output through display 30may be augmented (or even replaced) with a non-visual display 31. Thenon-visual display 31 can include tactile displays based on internalactuators, auditory feedback, or even displays based on conformalchanges in device appearance. For example, one possible feedback displayis based on internal auditory speakers (emitting a range of sounds fromsimple "beeps" to well formed speech, depending on available processorspeed and functionality) for providing user feedback. As will beappreciated, non-visual display 31 and its associated actuators orelectronics can support alternative feedback modes, including, forexample, force feedback to a user through internal actuators, tactilebased feedback (e.g with multiple surface projections for presentationof Braille or other conventional tactile user interface), modificationsto the surface texture of the device, or any other conventionalmechanism for supplying information to a user.

To better appreciate operation of the present invention, some selectedmodes of physical manipulation of device 10 are schematically presentedin FIG. 1. As seen in FIG. 1, device 10 can be translationally moved inthree spatial dimensions, as illustrated with reference to orthogonalforce arrows 50, 51, and 52. In addition to translational motion, thedevice 10 can be rotationally moved in any or all three spatialdimensions, as represented by arrows 53, 54, and 55. By use of sensorsystem 28 (alone or in combination with communication system 32), therelative or absolute position and orientation in three dimensions can bedetermined.

In addition to determining spatial position and orientation through useof sensors 28, the device 10 can optionally use sensor mesh 22 tomeasure and localize transient or continuing force application, withforce vectors and related timing information being determined andinterpreted. A few possible force actions (deformation modes) areschematically illustrated in FIG. 1, with arrows 60 and 61 illustratingdepression of surface 20 (with the combination representing a squeeze),arrows 62 and 63 illustrating sliding or rubbing deformation (with thecombination representing a twist), and the combination of sliding arrows65 and 66, and outward pull 67, together representing a pinch andoutward pull. The strength of the applied force can be measured (e.g. ahard or soft squeeze is differentiated), its spatial expanse found (e.g.to differentiate between poking with a fingertip or a thumb pad), andtiming determined (e.g. to differentiate between a quick or a slowdepression of the surface). The deformation so caused may be eitherpermanent or transitory.

As will be appreciated by those skilled in the art, each of theforegoing force actions represented by arrows can be considered asenseme. Some temporally distinguishable sensemes (or combinations ofsensemes such as the foregoing discussed pinch/pull combination) furtherrepresent morphemes used as a basis for a morphemic grammar inaccordance with the present invention. All of the following describedmorphemes can be modified by numerous variations in applied pressure,force utilized, appendages used, body parts, or extraneous intermediaryobjects used to apply force. In addition, timing of various objects(whether quick, slow, or alternately quick and slow) can modifyinterpretation of a morpheme. For example, if "squeeze" is taken as atypical morpheme, one can appreciate various squeeze operations such asquick squeeze, slow squeeze, hard squeeze, soft squeeze, narrow squeeze,wide squeeze, squeeze between hands, squeeze between one hand and auser's chest or head, squeeze between one hand and a table or wall, asqueeze made between two pens or two books, or even a squeeze between auser's tongue and the roof of the mouth. For purposes of the presentinvention, all such squeeze morphemes would be considered members of the"squeeze" class, with individual variations acting as possible modifiersor selected cases, just as a "house cat" might be considered aparticular member of the class of "felines", which more generallyembraces lions, tigers, and bobcats.

To aid in understanding the diversity of contemplated physicalmanipulation morphemes, FIG. 2 illustrates selected morphemes arrangedby increasing plasticity of the device required to enable morphemeutilization, and by increasing complexity of available senseme tuplesrequired to form or interpret a morpheme applied to a particular classof device. Beginning with the least plastic device and the simplestsenseme set used to compose a morpheme, a definition of a possiblephysical manipulation and typical function invoked by that manipulationof a device similar (but of course possibly more complex) to thatdescribed in conjunction with FIG. 1 is presented:

DEPRESS

Definition: Indenting one or more subregions of the device through theapplication of pressure.

Example: As seen in FIG. 3, consider a device 122 having a display 123.The device 122 supports a graphical illustration software applicationwhich allows users to position geometric shapes or user definedgraphical objects. The device can have four pads 124 placed around it,one on each side. By deforming a particular side, the user indicates adesire to "nudge" the currently selected geometric shape 125 away fromthat side to a new position 126.

SQUEEZE

Definition: To deform one or more subregions of a device through theapplication of vectors of force, some components of which are directedtowards each other, which compress the morphological structure of thedevice. This is a special case of depression.

Example: As illustrated in FIG. 4, consider a device 132 which candisplay one or more documents in either iconic (closed) or textualdisplay (open) modes. By selecting an open document 135, and thensqueezing a deformable edge 134 of the device 132, the user indicates adesire to make the document "smaller", in this case to iconize it asicon 136.

FOLD

Definition: To deform a second subregion by bending the first subregionsuch that it partially or completely overlaps the second subregion.Further deformations can be applied to that new morphological structureon other subregions.

Example: As illustrated in FIG. 5, consider a device 142 which candisplay documents. Suppose this device 142 has been augmented such thata deformable horizontal "flap" 144 on a top edge of the device 142 canbe folded to partially obscure a display 143. When the user makes thisfolding gesture, the user indicates a desire to password-protect("hide") the currently displayed document.

CURL

Definition: Deforming one or more subregions of the device by spirallyconfiguring subregions relative to each other in a cylindrical orcircular morphological structure.

Example: As illustrated in FIG. 6, consider a device 150 which candisplay documents in various languages (English, French, etc.). When theuser takes such a device 150, curls in a direction indicated by arrow157 into a tube, and then uncurls it, this "abracadabra" gesture tellsthe device 150 to display the current document in a different language.

STRETCH

Definition: Deforming one or more subregions of a device through theapplication of vectors of forces, some components of which are directedaway from each other, the vectors being applied at opposite ends of thedevice.

Example: As illustrated in FIG. 7, consider a device 160 with agraphical software application which allows users to manipulategeometric shapes. By stretching the device 160, the user indicates adesire to "resize" or "rescale" the currently displayed shape 165 to alarger size 166, the amount of resizing being a function of the amountof deformation. Note that SQUEEZING can indicate resizing the currentlydisplayed to a smaller size.

PINCH

Definition: Manipulating one or more subregions by applying vectors offorces, aligned directly towards each other, on opposite sides of theaffected subregion(s). This is typically, but not exclusively,accomplished using two-finger tactile force. PINCH is a special case ofSQUEEZE.

Example: As illustrated in FIG. 8, consider a device 170 which can copydocuments. By performing a "pinching" action 175, the user indicatesthat they wish the next set of copies to be issued in stapled form.

DOGEAR

Definition: Deforming a second subregion by folding a first subregion ata logical corner or edge of the second subregion, indicating a marker orlocation point (e.g. a bookmark) to be later referenced. DOGEAR is aspecial case of FOLD.

Example: As illustrated in FIG. 9, consider a device 180 which displaysa subset of the pages from a multi-page document. By "dogearing" theupper right corner 185 of device 180, the user indicates that they wisha bookmark associated with the currently displayed page or pages.

TWIST

Definition: Deforming one or more subregions of the device through theapplication of two opposing rotational forces offset from each other bya non-zero difference about some central axis.

Example: As illustrated in FIG. 10, consider a device 190 which, overtime, becomes degraded in some aspect of its performance (its diskbecomes fragmented, its memory needs garbage-collection, etc.). Byperforming a "TWIST" gesture 195, the user indicates that they wish thedevice to "wring itself out", performing, for example, garbagecollection.

RELIEF-MAP

Definition: Deforming one or more subregions of the device by raisingand or lowering them by either spatial transformation or theaddition/removal of material.

Example: As illustrated in FIG. 11, consider a device 200 which candisplay documents in either one- or two- page format. When the user"scores" the device by making a vertical indentation about the centeraxis of the device while it displays a single page 206, the device 200interprets the morpheme to request display of documents in a two-pageformat as pages 207 and 208.

RIP

Definition: Deforming one or more subregions of the device byintroducing a spatial discontinuity, by applying vectors of force topartially or totally disconnect these subregions from the device.

Example: As illustrated in FIG. 12, consider a device 210 which can copysome or all of its information. When the user performs the "ripping"gesture 215, removing one or more subregions, the device 210 copies itscurrently selected data set onto those subregions 216 and 217.

PERFORATE

Definition: Deforming one or more subregions of the device by means ofintroducing a change in the spatial connectivity of the subregions suchthat a hole is introduced (either temporarily or permanently) in thedevice.

Example: As illustrated in FIG. 13, consider a device 220 which is usedto route messages between various parts of a device network, and whichdisplays this functionality to the user by means of lines 222representing message pathways. When the user perforates the device 220with a finger or object 224 to introduce a hole in one of these pathways222, the system stops routing messages along that pathway.

SIMILARITY

Definition: Spatial motion, and/or physical deformation of one or moresubregions, of a device previously configured to represent some otherpredefined object or system. Typically the device acts in a mannerconsistent with the behavior of the real-world object when it ismanipulated in this manner. Such avatars, talismans, or othercomputationally enabled devices present themselves to a user as aphysical instantiation of another object or system. Avatars can bezoomorphic (including animal or human shapes) or configured to mimicreal world objects with any desired degree of specificity.

Example: As illustrated in FIG. 14, consider a device 230 capable oftext-to-speech audio output, audio input, and visual media presentation.The device 230 presents itself to a user in an anatomically accurateshape of a human head. When the user opens lips 232 on the human head,internal sensors detect opening of the lips and activate text-to-speechcapability. When the user covers or closes the eyes 234, associatedcameras or video systems are disabled, while covering ears 236 resultsin disabling audio recording systems.

3D MAP

Definition: The morphing of a device that can be molded around anexternal object to permit a sensor mesh to determine simultaneously sizeand shape of the external object The range of possible external objectsis large but is limited by the size of the device's solid inner-housingand the volume of the moldable material in the outer housing. In thissystem the device has the ability to accurately sense the amount ofmaterial from its inner surface to the outer edge of the moldablematerial (example: via ultrasonic sounding, similar to sonar in water)thus determining an accurate electronic model for the shape of themolded enclosure.

Example: As illustrated in FIG. 15, by pressing a device attachedmoldable material 244 around the surface of an external object (e.g. acog 242) a device 240 can automatically generate a CAD model of thatobject and store it in its memory.

MIMICRY

Definition: Deforming one or more subregions of the device such that theresultant morphological structure resembles a known real-world objectand through this association of subregions, the device acts in a mannerconsistent with the object it resembles.

Example: As illustrated in FIG. 16, consider a device 250 containing acomputer which contains text-to-speech and audio input capability, andwhich presents itself to the user as a misshapen blob having theconsistency and plasticity of moldable putty or clay. When the userperforms the "MIMICRY" action by molding part of the device 250 toresemble an ear, the audio input capability is activated.

In addition to morphemes based on physical manipulation, variousmorphemes based on varying degrees of relative or absolute spatialpositioning are contemplated to be useful in practice of the presentinvention. To aid in understanding the diversity of contemplated spatialmorphemes, FIG. 17 illustrates selected spatial morphemes arranged byincreasing knowledge of spatial position required to enable morphemeutilization, and by increasing complexity of available senseme tuplesrequired to form or interpret a morpheme applied to a particular classof device. Beginning with a device having only rudimentary relativepositioning functionality for supporting simple spatial sensemes, andending with a device absolutely positionable to within centimetersanywhere on Earth, a definition of a possible spatial manipulation andtypical function invoked by that manipulation of a device similar (butof course possibly more complex) to that described in conjunction withFIG. 1 is presented:

TRANSLATE (relative to device)

Definition: The linear movement of a device's center of mass from oneposition in space to another.

Example: Used to substitute for mouse controlled graphical "sliders" inconventional graphical user interfaces. When it is only physicallypossible to display a small amount of list, large listings can still besearched by "scrolling" a display window in response to the TRANSLATEmorpheme.

SHAKE

Definition: Spatially translating all subregions of a device by repeatedmovement in opposing directions, such that the net translation isnegligible.

Example: As illustrated in FIG. 18, consider a device 260 which is usedas a calculating device. When the user performs the "SHAKE" gesture, thedevice 260 clears its accumulator.

REVOLVE

Definition: Rotating all subregions of a device by rotating thesubregions about a point internal to the device, about any arbitraryplane.

Example: As illustrated in FIG. 19, consider a device 270 which displaysan imaged slice of volumetric data, such as medical data from a seriesof CAT scans. By rotating the device about a center point 272 internalto the device 270 to a new position 274, the plane specifying the imagedslice is changed accordingly.

TILT

Definition: Rotating one or more subregions of a device by rotating thesubregions such that one or more components of the rotary force are inthe direction of gravity and the amount of rotation is between about-180 degrees and +180 degrees.

Example: Consider a device which displays frames from an animationsequence on the side facing the user. As the device is tilted away, thespeed of the animation increases--as its tilted towards the user, thespeed of the animation decreases, analogous to operation of a gas pedal.

FLICK

Definition: A forwards TILT immediately followed by an opposingbackwards TILT.

Example: As illustrated in FIG. 20, consider a device 280 which cantransmit some subset of its data to another device. When the userperforms the "FLICK" gesture by quickly tilting in the direction ofarrow 282, followed by a reverse tilt along arrow 282, the device 280performs this transmission, towards the device (not shown) pointed to bythe ray of the gesture.

SPIN

Definition: Rotating one or more subregions of the device by rotatingthe subregions about a point internal to the device, such that the planeof rotation is one of the device's surface planes. SPIN can beconsidered a special case of

REVOLVE.

Example: As illustrated in FIG. 21, consider a device 290 which candisplay a frame of video 295 from a video sequence. The user performsthe "SPIN" gesture in a counter-clockwise direction, the device displaysearlier frames in the sequence; when the gesture is performed in aclockwise direction (arrow 292), the device 290 displays a later frame296 in the sequence (represented by film strip 294).

ORIENT

Definition: Rotating one or more subregions of the device by rotatingthe subregions about the center of the device, such that the plane ofrotation is one of the device's surface planes, and the amount of therotation is a multiple of 90 degrees (i.e. to rotate the device betweencardinal compass points). Orient can be considered a special case ofSPIN, which in turn is a special case of

REVOLVE.

Example: As illustrated in FIG. 22, consider a device 300 which candisplay a document in either 1 page, 2-page, or 4-page format ("1-up","2-up", or "4-up"). When the user performs the orient gesture in aclockwise direction (arrow 302), the device 300 increases the number ofdocument pages it is displaying from one page 305 to two pages 306 and307. Further orient gestures would increase the number of displayedpages. When performed in a counter-clockwise direction, the device 300decreases the number of pages it is displaying.

FACE

Definition: Manipulating one or more subregions of the device such thata first set of subregions is no longer bottom-most and a second distinctset of subregions now assumes the first subregions former position.

Example: As illustrated in FIG. 23, consider a device 310 which displaysdocuments, and which allows users to edit such documents. Considerfurther the case in which the device presents itself to the user in theform of a cube, in which 6 different documents are displayed on the 6different faces. When the user performs the "FACE" gesture by making aparticular face top-most, the document which is now on the top-most facebecomes editable by the user, while the document which is no longertop-most is no longer editable.

LIFT

Definition: The movement of a device's center of mass in a directionopposite to the current gravitation force acting on the device.

Example: Commanding the device to display the computer's file system ata position one level higher in the hierarchy.

PAN

Definition: The application of a TRANSLATION to a device such that it ismoved parallel to the front of the user's body, at a substantiallyconstant height.

Example: Viewing a spread-sheet on a device with a display so small onlyone cell can be shown. By PANing the device, the contents of the currentrow can be shown in sequence depending on the rate or amount of PAN.However, if the device were rotated away from its current orientationduring the PAN, a new row would be chosen. The selection of the rowcould be dependent on the deviation from the original orientation.

PUSH-PULL

Definition: Manipulating one or more subregions of the device byspatially translating them such that they are moved along a line ofprojection from the center of the device to the vertical axis of theuser's body.

Example: Consider a device with audio output capability. As the deviceis "pushed" further away from the body, its audio output levelincreases. When it's "pulled" towards the body, its level decreases.

WHACK

Definition: The application of an accelerative or de-accelerative forceto one or more subregions of a device such that said subregions contactor are contacted by an external object, causing an equal and oppositecountering force.

Example: As illustrated in FIG. 24, consider a device 320 which canperform long and unpredictable database searches. When the user performsthe WHACK gesture (e.g. upon a table 322), the current search isaborted.

ORIENT (relative to environment)

Definition: Manipulating two subregions of the device such that the linedrawn between the centers of those two subregions alters its orientationwith respect to the surrounding environment.

Example: Displaying a CAD drawing of a machine-part in 3D on the displayof a mobile device. As the orientation of the device changes, so toodoes the viewing angle and position of the rendered image.

ORBIT

Definition: Rotating one or more subregions by rotating the subregionsand/or the center of mass of the device about some point exterior to thephysical boundaries of the device, about any arbitrary axis.

Example: As illustrated in FIG. 25, consider a device 330 which cansearch a network database, such as the World Wide Web, for information.When the user performs the "orbit" gesture, such a search is initiated.The radius 332 of the rotation 334 specifies the breadth of thesearch--wider circles specify a wider search. The speed of the gesturespecifies the time limit imposed on the search--the quicker the gesture,the more cursory the search.

ORBIT RELATIVE TO USER

Definition: Rotating one or more subregions by rotating the subregionsand/or the center of mass of the device about some point exterior to thephysical boundaries of the device, where said point is proximal to abody feature of the user. This is a special case of ORBIT.

Example: As illustrated in FIG. 26, consider a device 340 which canperform audio output. By performing the ORBIT gesture (in directionindicated by arrow 344) about the user's ear 345, the audio output isactivated.

MOVEMENT IN ROOM

Definition: The local detection of a device's 3D position relative toreference points found within an enclosing room. Differences in themeasured position are used to trigger actions.

Example: A virtual filing system that allows you to save and restorefiles based on the devices current position in the room. To save a fileyou might think carefully about the contents of the file and then walkto the position in the room that might be most easily associated withit. When retrieving the file you would use the same thought processesand go back to the position you had associated with the file. On doingso the files associated with that position would be displayed and youwould then be able to select the file you were looking for. The systemis useful because the human mind is very good at remembering informationthat is spatially organized rather than in some abstract informationdata structure.

MOVEMENT BETWEEN WIDELY SEPARATED SITES

Definition: Manipulating one or subregions of the device such that thedetected absolute spatial position of those subregions is changed.

Example: Consider a device which can display information from a databaseof client information. When the device is moved to a different clientsite, the device automatically updates its display to displayinformation for the nearest client site.

In addition to morphemes based on physical manipulation or spatialpositioning, various morphemes based on sensed environmental conditionsare contemplated to be useful in practice of the present invention. Toaid in understanding the diversity of contemplated environmentalmorphemes, FIG. 27 illustrates selected environmental morphemes looselyarranged in order of increasing sensor complexity needed in somecommonly sensed environmental categories. For each of the profferedcategories, some selected sensing systems supportable by a device suchas that described in conjunction with FIG. 1 are presented:

LIGHT

Definition: Manipulating one or more subregions of the device such thatthe amount of light falling upon those subregions changes.

Example: Consider a device used in a lecture hall to take notes. Whenthe room lights are turned on, the light sensors detect this and turndown the backlight to conserve energy. When the room lights are turnedoff (e.g. during a slide show) the light sensors detect this and turn upthe backlight to increase viewability.

Light sensors can range from thresholded binary light detectors, tolight pattern detectors, to full imaging systems. Advanced techniquescan include image analysis and recognition to identify objects orpersons.

HEAT

Definition: Manipulating one or more subregions of the device such thatthe amount of heat applied to those subregions changes.

Example: Consider a portable computer which has a stylus for enteringtext. By looking at the heat profile along the back surface of thecomputer, the computer can detect whether it is being held with the lefthand, the right hand, both hands, or neither hand, and update itsinterface accordingly.

Thermal (heat) sensors can range from simple temperature sensors tosophisticated differential thermal mappers and thermal imagers.

ELECTROMAGNETIC

Definition: Manipulating one or more subregions of the device such thatthe electromagnetic spectrum applied to those subregions changes.

Example: By analyzing the radio spectrum, the device can deriveestimates as to its absolute spatial position, and use that to alter itsfunctionality.

Electromagnetic detection can include magnetic compasses, radiodetection, or GPS signal detection. More advanced techniques can includeelectromagnetic spectrum analysis and interpretation, such as roughlydetermining location based on available radio signals.

VIBRATE

Definition: Manipulating one or more subregions of the device byvibration.

Example: Consider a device which displays textual information. When theuser takes the device on the bus, the ambient vibration level sensed bythe device changes, and the device increases the size of the displayedtext to help the user compensate.

This class of environmental morphemes can include detection ofintermittent contacts, low frequency rumblings, or acoustic leveldetection. More advanced techniques requiring greater processor powerinclude maximum frequency identification, spectral analysis of acousticfrequencies (enabling the device to distinguish background environmentalnoises from speech, for example), or even speech based identification ofpersons in the vicinity of a device.

In addition to morphemes based on physical manipulation, spatialposition, or sensed environmental factors, various morphemes based oncooperation between multiple interacting devices are contemplated to beuseful in practice of the present invention. To aid in understanding thediversity of contemplated spatial morphemes, FIG. 28 illustratesmultidevice morphemes arranged by an increasing level of possiblephysical contact, and by increasing complexity of available sensemetuples required to form or interpret a morpheme applied to a particularclass of device. Beginning with a device having only rudimentary edgedeformation functionality for supporting simple spatial sensemes, andending with complex deformable or embeddable devices that can be wrappedabout each other, a definition of a possible multidevice manipulationand typical function invoked by that multidevice manipulation of devicessimilar (but of course possibly more complex) to that described inconjunction with FIG. 1 is presented:

TOUCH

Definition: To move one or more subregions of a device such that theyenter physical contact with a subregion of a second device, in anyalignment and to any extent. Or, to take two devices so aligned andremove that alignment.

Example: Consider two portable computers 350 and 351, the first of whichcontains a database, and the second of which contains an IRDA port. Whenthe user touches the first computer 350 to the second computer 351, thedatabase is transmitted via the second computer's port.

MATCH

Definition: To move one or more subregions of a device such that theyenter physical contact with one or more subregions of a second device,with the subregion(s) of the first device and the subregion(s) of thesecond device being aligned along one or more edges. Or, to take twodevices so aligned and remove that alignment.

Example: As illustrated in FIG. 30, consider, multiple devices 360, 361,and 362, which contain different versions of the same basic database.When the user MATCHes the first device 360 to the second device 361,followed by matching the third device 362 to the matched first andsecond devices, their databases are reconciled (synchronized).

STACK

Definition: To move one or more subregions of a device such that theyenter physical contact with a subregion of a second device, such thatthe first device is now located above, but physically adjacent to, thesecond device. Or, to take two devices so aligned and remove (i.e.unstack) that alignment.

Example: As illustrated in FIG. 31, consider a set of devices 370, 371,and 372, each of which is displaying a frame of video from a longervideo sequence. When the devices are stacked, the ordering of thestacking specifies an order for the video editing, and a singlecomposite video is now produced.

TILE

Definition: To move one or more subregions of a device such that theyphysically contact with a subregion of a second device, such that thefirst device and second device now form a single seamless spatial unitor to take two devices so aligned and remove that alignment. TILE is aspecial case of MATCH.

Example: As illustrated in FIG. 32, consider a set of devices 380, 381,382, 383, each of which can independently display a portion of a largephotograph. When the devices are tiled, each device displays the portionof the photograph appropriate to its current relative position in thetiled grid.

RELATIVELY ALIGN

Definition: To move one or more subregions of a device such that theyengage in a particular spatial relation to one or more other devices,where said devices are not touching.

Example: As illustrated in FIG. 33, consider a set of devices 390, 391,392, and 393 which are displaying a multi-page document. Whicheverdevice is presently placed at the far left (device 390) displays thetable of contents, whichever one is presently placed at the far right(device 393) displays the index, and the others display pages accordingto their respective locations. As different devices can have differentdisplay capabilities, moving them about can alter the document display.For example, if only one of the devices has a color display, when it ismoved from second position to third position then (a) the device whichwas in third position, displaying page #2, now displays page #1, and (b)the color display, which was displaying page #1, now displays page #2 incolor.

WRAPPING/EMBEDDING

Definition: Manipulating one or more subregions of a device such thatthese subregions spatially occlude or are spatially occluded by someportion of the second device.

Example: As illustrated in FIG. 34, consider a first device 400 whichcontains the infrastructure to support the filtering of email. Considera second set of devices 401 and 402 which implement particular emailfilters. When the user physically embeds device 401 (or 402) into thefirst device 400, by which the first device 400 now wraps the seconddevice 401 (or 402), the particular email filter supported by the seconddevice is activated.

As those skilled in the art will appreciate, combinations of any of theforegoing described morphemes based on physical manipulation, spatialposition, environmental conditions, or multiple interacting devices canbe extended by participation in a morphemic "sentence". A sentence isdefined as a sequence of one or more temporally disjoint morphemes.Typically, between about 1/10th of a second and 2-3 seconds suffices todistinguish morphemes within a sentence. Of course, in somecircumstances and indefinite time period may elapse. The sentence levelallows definition of a physical manipulatory grammar by appropriatechoice of a morpheme sequence, and corollary rules governing, forexample, use of active (verb like) morphemes, naming (noun) morphemes,or connectors. Just as the position and relation of words in a sentencedefine the sentence's meaning (e.g. "horse chestnut" is not the same as"chestnut horse"), similarly the position and relation of morphemes in amanipulatory sentence define the sentence's meaning. For example, in acommunication mode a FLICK followed by a WHACK could mean "transfer dataand erase the local copy", while a WHACK followed by a FLICK means"power on the device and transfer data". In other settings, a FLICK or aWHACK could mean something entirely different. To better understandconstruction of morphemic sentences, the following examples arediscussed:

DATA TRANSFER SENTENCE

Consider a device which can transmit some or all of its information toanother device. Furthermore, this transmission can be done eitherunencrypted, or encrypted (to increase security). Furthermore, thistransmission of a text+graphics document can either include thegraphics, or omit them (to save time). Suppose that the user wishes toperform the command "Transmit the information in document A, encrypted,omitting graphics, to machine B". Then a gestural sequence (morphemicsentence) to support this could be:

DEPRESS--the user presses on a displayed representation of A, indicatingthat A is to be selected for an upcoming operation

FLICK--the user flicks the device in the direction of device B,indicating that the operation is a transmission to B

FOLD--the user folds the top quarter of the device over the lowerthree-quarters, indicating that the transmission is to be encrypted.

TWIST--the user twists the device about its central axis, indicatingthat the data is to be "wrung out", i.e. the graphics are to be omitted

SQUEEZE--the user squeezes the device, indicating that they areconfirming that the operation is to proceed.

Note that none of these gestures, in isolation, performs an act--the"ensemble" of temporally separated morphemes must be interpreted inorder to form the complete action.

DRAWING MODIFICATION SENTENCE

Consider a device which displays geometric shapes for user manipulation.Furthermore, one of the manipulations supported is to resize (orrescale) a shape. Furthermore, suppose that this resizing can be doneeither aliased Baggy) or anti-aliased (edges are smoothed). Suppose thatthe user wishes to perform the command "Resize shape A by 120%, aboutthe X axis only, using anti-aliasing". Then a morphemic sentence tosupport this could be:

DEPRESS--the user presses on a displayed representation of A, indicatingthat A is to be selected for an upcoming operation

STRETCH--the user stretches some portion of the device, indicating thatthe operation is to be a resize. When the user starts stretching, aportion of the status display displays "100". The user continues tostretch until the status display reads "120"

RELIEF-MAP--the user "scores" the display by making a horizontal line ofdepression, indicating that the operation is to take place only aboutthe horizontal (X) axis.

DEPRESS--a circular thumb stroke is made in a different area of thedevice, indicating that anti-aliasing (smoothing the edges) is to beperformed.

DATABASE PRESENTATION SENTENCE

Consider a device which contains various personal information databases,such as a list of phone numbers, a list of addresses, and a calendar.Suppose that the user wishes the most appropriate of those databasesdisplayed. Then a gestural sequence to support this could be:

SPATIAL LOCATION--the user carries the device such that it is spatiallyproximate to either the telephone, the address book, or the refrigerator(where the family calendar is displayed), whichever is appropriate.

DEPRESS--the user touches the device to activate the operation. Thedevice now displays the personal information appropriate to thatlocation.

DATABASE RETRIEVAL SENTENCE

To extend the foregoing example of a database presentation sentence,consider two computers which contain calendar databases. If the userwishes to synchronize the calendars, a suitable gestural sequence tosupport this could be:

SQUEEZE--the user squeezes the device to activates its gesturalrecognition capabilities

ORBIT--the user orbits the device about the surface of the other device3 times, indicating a desire to only match data for the next 3 weeks.

MATCH--the user matches the edge of the device to the edge of thecalendar, indicating a desire to "match" contents between the twodevices.

PRINTER/COPIER CONTROL SENTENCE

Consider a device which can produce paper copies of documents. Supposethat the user wishes to tell such a device to produce a stapled, twosided copy of document A, enlarged to the next greater size. Then agestural sequence to support this could be:

DEPRESS--the user presses on a displayed representation of A, indicatingthat A is to be selected for an upcoming operation

RIP--the user introduces a spatial discontinuity into a portion of thedevice, indicating that the upcoming operation is to be a copy("carrying away" some of the data).

PINCH--the user pinches the upper left corner of the device, indicatingthat the copies are to be stapled.

SQUEEZE--the user presses on the front and back of the device,indicating that the copy is to be two-sided.

STRETCH--the user stretches the device, indicating that the copy is tobe an enlargement to the next greater size.

FACE--the device typically has its paper emitter on the bottom,preventing users from making copies accidentally. By facing the devicesuch that the emitter is on the side, the copying operation isinitiated.

LIGHT BASED CONTROL SENTENCE

Consider a device which can display documents. Suppose the user is usingthe document while seated on a train, and wishes the document to displayitself with a backlight when the train enters a tunnel, and wishes thedocument to display itself in a larger font when the train rumbles overrough tracks. Then a gestural sequence to support this could be

SQUEEZE--the user squeezes on the device, indicating that a loss oflight is to be compensated for by a backlight.

LIGHT--as the train enters a tunnel, the LIGHT gesture is made, and thedevice turns on the backlight.

WHACK--the user briskly raps the device against the palm of their hand,indicating that their desired preference for adjusting hard-to-readdocuments is to increase the font size.

VIBRATE--as the train goes over a bridge, the vibratory gesture issensed. Because of the position of this VIBRATE morpheme (after thepreceding WHACK gesture) in this morphemic sentence, the device nowincreases the font size on the displayed text.

LIGHT--the user puts the device into his suitcase, making the LIGHTgesture. In this context (with no SQUEEZE before it), the LIGHT gesturecauses the device to power-off its display.

To better appreciate utility and construction of devices in accordancewith the present invention, several examples of devices are nowdescribed:

PORTABLE COMPUTER WITH SQUEEZE AND TILT CONTROL

A handheld portable computer 500 (e.g. a 3Com® PalmPilot®) capable ofbeing fitted with deformable, pressure sensitive edging 504 isschematically illustrated in FIGS. 35 and 36. The computer 500 supportsa name and address software application, providing a user viewablename-address entry field on display 503. In this embodiment, a user cansqueeze the deformable, pressure sensitive edging 504 (squeeze arrows507) of the computer 500. In response, the name and address softwareapplication causes the display 503 to animate by slowly incrementing(scrolling) through the name list from "A" towards "Z" entries. When theuser squeezes edging 504 again, the software application stops thescrolling animation. Scrolling functionality is further enhanced by theuse of a tilt sensor, which allows the computer's behavior to mimicconventional rotatable address books. If the computer 500 is tilted awayfrom the 45 degree angle at which someone might typically hold it, thescrolling rate is increased. In this application, the closer thecomputer 500 was tilted towards the user (as indicated by arrow 506 ofFIG. 36), the faster the scroll rate toward "Z". However, if a usertilted the computer 500 back past the neutral 45 degree position (asindicated by arrow 506 in FIG. 36), the animation would move backwardswith a velocity related to the magnitude of tilt. In this way it waspossible for a user to search for items in a long list in a very naturalway, while only using one hand.

In an alternative mode schematically illustrated in FIG. 37, scrollingspeed can be completely controlled by pressure. The greater the squeezepressure (arrows 537), the faster the list scrolls. Release of theapplied pressure causes the scrolling to halt. In this alternative userinterface strategy, application tilt (as indicated by orthogonal tiltarrows 530 and 532) could be used to change the direction of thescrolling through the list, allowing a user to search portions of alarge two dimensional data set (schematically illustrated as a dataplane 520) that is not entirely visible either horizontally orvertically on display 503. By simply tilting the display 503 of computer500 as if it were a window through which the data plane 520 can beviewed, any particular portion of the data plane (for example, datasubset 524) can be viewed. As will appreciated, in both the foregoingmodes the speed of scrolling, the specific neutral tilt angle, andrequired pressures to initiate scrolling changes can be adjusted to fita particular user.

PORTABLE COMPUTER WITH HANDEDNESS DETECTION

Pressure sensors have been added to augment a conventional keyboard 551enabled user interface to a hand holdable Windows® CE class computer 550(i.e. a Cassio® Cassiopia®) schematically illustrated in FIGS. 38 and39. In this embodiment, user handedness was determined by using pressuresensors positioned on a right back-edge and a left back-edge of computer550. User studies have found that the difference in pressure between theright and left side gave a direct indication of handedness of a user. Asrespectively illustrated in FIGS. 38 and 39, handedness was used tojustify formatted text 554 to the left (FIG. 38) or right (FIG. 39),thereby allowing more space 555 on display 553 for an electronicannotation pen to be used to mark-up the text.

For both the embodiment of the invention illustrated by FIGS. 35-37, andthe foregoing embodiment illustrated in FIGS. 38 and 39, materialdeformation of a spongy, elastic, or otherwise deformable material mustbe measured. Although various techniques can be used to measure materialdeformation, including those based on imaging or fluid volumetricchanges, one particularly useful technique is based on the use ofpressure transducers. Commercially available sensors measure pressure(indicative of material deformation) by converting a pressure change toa change in electrical characteristics. For example, inexpensive sensorsthat change resistance in response to pressure can be obtained in avariety of shapes and sizes, including paper thin sensors, and easilybendable sensor strips. Sensors of this kind can be customized to anyparticular shape or form that a gestural Ul might require. The change inresistance is usually linearly related to pressure, with the sensorgenerally being placed in a potential divider network to model thepressure as a change in potential. For a practical circuit the resultingsignal needs to be amplified, buffered and translated such that thechange in value from minimum pressure to maximum pressure spans a usefulrange. The modified signal can now be fed into an analog to digitalconverter (ADC) to produce a digital representation of pressure. An8-bit ADC can typically be used for most applications, however, ifgreater sensitivity to pressure changes is required, a higher resolutionADC (e.g 16-bit ADC) can be used. As will be appreciated, the ADC couldbe memory mapped into the processor's address space as a peripheral, oralternatively supplied as a retrofitted pressure interface to existingcomputers capable of benefiting from this system. Since an RS232connection is an interface that is almost universally available onportable computers, one strategy is to have the parallel output of theADC converted into a serial RS232 frame using a serializer such as aUART and then level-shift and buffer the signal as specified by theRS232 standard. At the computer end of the serial interface, anotherlevel-shifter and UART, the output of which is readable by theprocessor, performs the serial-to-parallel conversion.

As described with reference to FIG. 40, in realizing a working system amicrocontroller 564 (a Signetics 87c752) with an ADC built in to combinemany of the I/O tasks in a single chip can be used in combination withlevel shifter 566 (a MAX3223). This approach has the advantage thatintelligent processing of the input signal is possible with software.Tilt measurement was provided by a tilt sensor 567 connected to abuffer/amp 562 to supply an analog signal to microcontroller 564. Thepressure measurement can also be encoded within a protocol across theserial link. This particular microcontroller 564 has five ADC inputs,but by making use of eight digital control lines, it is possible to useonly one ADC input and one buffer amplifier 561, to measure up to eightpressure points with pressure sensors 565. This is achieved by using thecontrol lines to select only one sensor at a time and take a reading foreach using a single input to the ADC. After eight sensors have beenselected, eight readings are acquired in memory. The design is practicalbecause the microcontroller can take measurements and make analog todigital conversions at a rate far higher than is needed to communicatewith a host computer 569.

For scrolling or handedness based software applications, 16 levels weredetermined to be adequate for representing a pressure measurement. Inorder to have a high data throughput to the host computer 569, eachmeasurement was encoded in one byte of an RS232 frame such that the fourlowest bits were the pressure representation and the highest four bitswere the sensor ID. Each frame of the RS232 data was thereforecompletely self contained. Of course any protocol that limits itself toa specific number of devices in its address space will eventually havethe problem that someday applications might be designed that need toreference far more devices than are supported. The solution used in thisprotocol is to reserve the sensor ID number 15 as a special value thatcan extend the semantics of the encoding to include an arbitrary numberof bytes representing a sensor or value. For the described softwareapplications, the commonly used RS232 frame format (1 start, 8 data noparity, 1 stop bit at a baud rate of 9600) was chosen.

In operation, the host computer 569 determined handedness by utilizinginformation about the current pressure exerted on two subregions--one,on the back of the device, occupying roughly the left half, and theother, also on the back of the device, occupying the right half. Thecurrent pressure values were converted from analog to digital form, suchthat a digital value of 0 (zero) represents no pressure, and a digitalvalue of, for example, 15 represents maximum pressure. The detectioncircuitry then proceeds:

if (left₋₋ sensor is high AND right₋₋ sensor is high) then conclude thatthe user is gripping the device with both hands

else if (left₋₋ sensor is high AND right₋₋ sensor is low) then concludethat the user is gripping the device with the left hand only

else if (left₋₋ sensor is low AND right₋₋ sensor is high) then concludethe user is gripping the device with the right hand only

else if (left₋ sensor is low AND right₋₋ sensor is low) then concludethat the user is gripping the device with neither hand

Also, to optimize communication, pressure values are only sent when theychange. In order to account for jitter and error in the pressuresensors, a sensor is only considered `high` if its value is higher thansome minimum threshold (e.g. `2` on the 0 to 15 range).

SCANNER/PRINTER/COPIER SUPPORTING EDGE DEFORMABLE DISPLAY

A scanner/printer/copier device 570 is schematically illustrated in FIG.41. As seen in FIG. 41 (and in more detail in FIG. 42) the device 570supports a display 574 having a deformable edge 572, with thecombination roughly shaped like a piece of paper. In operation, a usercan place a written document in the device 570 for scanning. Anelectronic version of the scanned document is displayed (i.e. as text575 in FIG. 42) on the display 574. By outwardly pulling the deformableedge 572 as indicated by arrow 577, a user can instruct the device 570to resize the document before printing or copying. Pinching opposingsides of the deformable edge 572 (arrows 578) can further instruct thedevice 570 to print double sided copies. As will be appreciated, variousother morphemes can be used to provide a simple interface forinteracting with device 570.

TANGIBLE DEVICE FOR COMPUTER LOGON

A password protected personal computer 612 is a component of a computersystem 610 that includes connection to a computer network 614. Thetangible computing device takes the form of a stylized key 620, inaccordance with a common metaphoric or analogical connection betweenaccess restricted physical security and a key. Like a conventional keyused to open a door, the stylized key 620 can "open" computer system 610by initiating logon procedures and providing passwords to a computer612.

Typically, a wireless acoustic, radio or infrared system is used topermit intercommunication between the key 620 and the computer 612. InFIG. 43, a conventional infrared transceiver 618 connected to computer612 is used to receive and transmit infrared signals 625 to an infraredtransceiver 624 supported by key 620. The infrared transceiver 624 isconnected to a processor/memory unit 626 positioned within key 620. Thekey 620 also includes processor/memory unit 626 connected sensor 622 andfeedback unit 628. The sensor 622 can be constructed to detect tilt androtation using conductive metal liquid switches, accelerometers, gravityor acceleration detecting microelectromechanical mechanisms, or anyother convenient mechanism for detected transient tilts or rotations ofthe key 620. The sensor 622 can also support pressure sensingmechanisms, squeeze or stretching sensors, thermal sensors, auditorysensors, or even complex bioidentification sensing units to determinefingerprints, voiceprints, or other unique identifiers of a user. Thefeedback unit 628 is used to generate user perceptible auditory,tactile, or visual feedback. This includes, for example, an audible"click", a vibrating element, a flashing light emitting diode, or anyother suitable mechanism for indicating state change in the computer612.

In operation, a user can log onto personal computer 612 by firstbringing the stylized key 620 into a vicinity of the personal computer612. The user simply rotates the key 620 (in the direction indicated byarrow 627) to cause the tilt/rotation sensor unit 622 to report arotation to the processor/memory unit 626. The processor/memory unit 626then sends (via infrared transceiver 624) various user identification,logon, and password information to computer 612 connected infraredtransceiver 618. The computer 612 responds with an acknowledgementsignal, causing the feedback unit 628 to generate, for example, a harshbuzzing sound if the password is incorrect or, if the password iscorrect and logon successful, a series of brief audible clicks similarto the sound generated by turning tumblers in a conventional mechanicallock and key system.

As those skilled in the art will appreciate, it is possible tosubstitute various other forms for the stylized key. A tangiblecomputing device having the form of a garage door opener could be used,or any other suitable form sufficiently large to support the necessaryprocessor, transceiver, and feedback electronics. In addition, variouscombinations of manipulation gestures such as previously discussed (e.g.squeezes, whacks, spins etc.) can be used as a substitute for a personalidentification number (PIN). For example, activation of the key 620 mayfirst require three quick squeezes or other user definable manipulationprior to tilting for login. This would provide an additional measure ofsecurity in case of loss or theft of the key 620, just as requiring aPIN number reduces misuse of bank cards.

ZOOMORPHIC DEVICE FOR COMPUTER LOGON

Zoomorphic computer devices, discussed briefly in relation to FIG. 1,are a particularly preferred form of tangible devices. Small animal orhumanoid shaped zoomorphic devices can help overcome some of theinterface disadvantages associated with large, abstract, and/or imposingrepresentations such as a human head (e.g. as seen in connection withFIG. 14), while still providing better recognizability than abstractshapes (e.g. cylinders or boxes). Like a tangible device, any zoomorphicdevice can represent either a user or a desired physical/virtual system,providing a correspondence in which the zoomorphic device canalternatively be known as an "avatar" or "talisman", since itsymbolically represents a user, a physical system, or a virtual system.

For example, FIG. 44 illustrates an alternative computer logon/passwordsystem that uses one or more zoomorphic devices in place of the tangiblestylized key of FIG. 43. A "guard dog" 640 is a zoomorphic avatar formedto have internal electronics and transceiver 624 substantially identicalto those discussed in connection with key 620 of FIG. 43. Like thestylized key 620, the guard dog can support pressure sensing mechanisms,squeeze or stretching sensors, thermal sensors, auditory sensors, orcomplex bioidentification sensing units to determine fingerprints,voiceprints, or other unique identifiers of a user. The guard dog canalso generate user perceptible auditory, tactile, or visual feedback toallow a user to determine various system states. Feedback includes, forexample, canine audio utterances such as barks and growls, orelectrically activated tail wagging.

In operation, a user can log onto personal computer 612 by firstbringing the guard dog 640 into a vicinity of the personal computer 612.After physically activating the guard dog 640 by a series ofmanipulations (e.g. squeezes 651 or rotations 653 that must be appliedin a secret order), a user can place the guard dog at a predeterminedposition 655 on a position sensitive surface 646. Once properlyactivated and positioned with respect to the surface 646, the guard dogsends (via infrared signal 625 generated by infrared transceiver 624)various user identification, logon, and password information to computer612 connected infrared transceiver 618. As previously discussed inconnection with FIG. 43, the computer 612 responds with anacknowledgement signal, causing the guard dog 640 to generate, forexample, a series of brief greeting barks or even electromechanicallyactivated tail wags (or movement of other appendages such as the guarddog's paw) if the password is correct and logon successful. When a guarddog is positioned incorrectly, such as illustrated with respect to guarddog 644 placed at position 657, the guard dog can emit a warning growl,rotate in place, move slightly to another place or position, or provideother suitable indication that the logon procedure has failed.

The use of secondary or auxiliary guard dogs that interact with otherguard dogs or the personal computer 612 is also contemplated within thescope of the present invention, For example, a "network" guard dog 642,having substantially identical functionality and electronics as guarddog 640, can be used to logon to network computer system 614 after guarddog 640 has enabled logon to personal computer 612. Guard dog 642 cancommunicate directly with the personal computer 612, or via wirelesslink to the guard dog 640. As will be appreciated, guard dogs (oralternative tangible computing devices or zoomorphs) can convey varioustypes of information to each other, to personal computers, or tonetworked computers. For example, guard dog 640 can convey user profileinformation, address lists, or any other desired information toassociated personal computers. In an alternative use, users can usetheir guard dogs to grant others having a suitable guard dog limitedaccess to personal or network data files or applications.

MEDIA SPACE PENGUIN

As seen in FIG. 45, a zoomorph shaped to resemble a penguin 720 havingvarious accessories (e.g. LCD screen 722 for status display, hat 730,ear mufflers 732, and darkened glasses 734) can be an interface to amedia control system 710 using a wireless infrared transceiver 724. Amedia control system 710 (alternatively known as a "media space")includes a personal computer 612, monitor 613, infrared transceiver 618,and network system 614 substantially identical to that discussedpreviously in connection with FIGS. 43 and 44. These interact with anaudiovisual server 716 connected through audio/visual switch 718 tovarious audiovisual input/output devices, including a camera 740,microphone 742, and audio speaker system 744. The penguin 720 hasvarious features or physical attributes that a user can symbolically ormetaphorically relate to the media control system 710. The presentinvention takes advantage of some typical symbolic or metaphoricallinkages to enable physical manipulation, orientation, or positioning ofthe penguin to be interpreted as commands to the media control system710.

For example, the penguin can be considered an avatar of the media space,and can be operated analogous to the head previously described inconnection with FIG. 14. For example, covering eyes of the penguin withthe darkened glasses 734 means the media space camera 740 is turned off,covering the ears means the media space turns off or disables themicrophone 742, closing the penguin's mouth (as indicated by arrows 727)mutes audio output from speaker system 744

As those skilled in the art will appreciate, it is also possible toconfigure the penguin to be an avatar of a user. In such a case, thesymbolic or metaphorical linkages are altered so that covering the eyesof penguin 720 means the user does not want to see the media space asdisplayed by monitor 713, covering the ears of penguin 720 means theuser does not want to hear the media space from speaker system 744, andcovering the mouth of penguin 720 means the user does not want to beheard, causing the microphone to be disabled.

USER NOTIFICATION ZOOMORPH

In addition to acting as a user interface for controlling physicalsystems, a suitably modified zoomorph such as penguin can be used as forauxiliary or primary notification and messaging services. As seen inFIG. 46, the penguin 720 can be used to notify a user of receipt ofelectronic mail (via network 614 to personal computer 612) by raisingits arm 728 (in the direction of arrow 729) using conventionalelectromagnetic mechanisms built into the penguin 720. Advantageously,this allows a user to determine useful information (the presence ofwaiting electronic mail) by the position of arm 728 without explicitlyaccessing a computer or viewing the monitor. Alternatively, acoustic orlight output can be used to signal receipt of electronic mail, alone orin addition to physical movement of the zoomorph. A sender's name,subject header, or even the mail message in whole or in part canoptionally be displayed on the screen 722 of the penguin 720, as well asbeing available on the monitor 613. As will be appreciated, notificationservices are not limited to flagging receipt of electronic mail, but caninclude notification or messaging for changes of state in a wide varietyof physical and virtual systems. Such systems may include, but are notlimited to people (a user himself or other people), environments,computational systems, or automated tracking systems. Systems may berepresented by one or more physical avatars, or alternatively, a singletangible device or zoomorphic avatar may represent one or more physicalor virtual systems.

As those skilled in the art will appreciate, other variousmodifications, extensions, and changes to the foregoing disclosedembodiments of the present invention are contemplated to be within thescope and spirit of the invention as defined in the following claims.

The claimed invention is:
 1. A zoomorphic computer user interfacecomprising,a computer system directable both to accept user input tomodify its system state, and to provide user output, and a zoomorphicunit having a recognizable zoomorphic shape and a deformable surface,wherein the zoomorphic unit is connected to the computer system toprovide user input and react to computer system output, and wherein thezoomorphic unit comprises at least one sensor mesh coupled to thedeformable surface to monitor deformation of the deformable surface. 2.A zoomorphic computer user interface comprising,a computer systemdirectable both to accept user input to modify its system state, and toprovide user output, a zoomorphic unit having a recognizable zoomorphicshape and a deformable surface, wherein the zoomorphic unit is connectedto the computer system to provide user input and react to computersystem output, and a reference unit connected to the computer system andavailable for interaction with the zoomorphic unit, with positioning ofthe zoomorphic unit with respect to the reference unit directablymodifying system state of the computer system, wherein the zoomorphicunit further comprises at least one sensor mesh coupled to thedeformable surface to monitor deformation of the deformable surface. 3.The zoomorphic computer user interface system of claim 2, wherein thezoomorphic unit further comprises at least one non-visual feedback unit.4. The zoomorphic computer user interface system of claim 3, wherein thenon-visual feedback unit of the zoomorphic unit further comprises atleast one movable appendage.
 5. The zoomorphic computer user interfacesystem of claim 2, wherein the zoomorphic unit further comprises atleast one visual feedback unit.
 6. The zoomorphic computer userinterface system of claim 2, wherein the zoomorphic unit furthercomprises at least one audio feedback unit.
 7. A zoomorphic computeruser interface comprising,a computer system providing input and output,a zoomorphic unit having a recognizable zoomorphic shape, at least onemovable element, and a deformable surface, the zoomorphic unit beingconnected to the computer system, with the zoomorphic unit having asensor mesh to detect deformation of the deformable surface and provideinput to the computer system, and a feedback unit for modifying positionof the movable element in response to computer system output.
 8. Thezoomorphic computer user interface of claim 7, wherein the at least onemovable element is an appendage.
 9. The zoomorphic computer userinterface of claim 7, wherein computer system has a messagingapplication for receipt of electronic messages, and wherein the at leastone movable element of the zoomorphic unit is movable in response toreceipt of electronic messages.
 10. The zoomorphic computer userinterface of claim 7, wherein the zoomorphic unit further comprises atleast one visual feedback unit.
 11. The zoomorphic computer userinterface system of claim 7, wherein the zoomorphic unit furthercomprises at least one audio feedback unit.
 12. The zoomorphic computeruser interface system of claim 7, wherein the zoomorphic unit furthercomprises a wireless transceiver for communication with the computersystem.
 13. A zoomorphic computer for controlling a computer systemproviding input and output, the zoomorphic computer comprising,azoomorphic shell having a recognizable zoomorphic shape and a pluralityof movable subregions, a transceiver attached to the zoomorphic shellfor communicating with the computer system, a position detecting unitfor determining position of the zoomorphic shell relative to thecomputer system, with change of position of the zoomorphic shellrelative to the computer system changing state of at least one of thezoomorphic shell and the computer system, and a sensor mesh in contactwith the zoomorphic shell, the sensor mesh detecting deformations withinthe movable subregions of the zoomorphic shell, and transmittinginformation related to multiple deformations within the movablesubregions to the transceiver to provide input to the computer system.14. The zoomorphic computer of claim 13, further comprising an audiofeedback unit.
 15. The zoomorphic computer of claim 13, furthercomprising a visual feedback unit.
 16. The zoomorphic computer of claim13, wherein position of the zoomorphic shell with respect to thecomputer system determines success of computer system logon.
 17. Thezoomorphic computer of claim 13, further comprising an audio/visualsystem connected to the computer system, and wherein manipulation of thezoomorphic shell controls state of the audio/visual system.
 18. Azoomorphic computer for controlling a computer system providing inputand output, the zoomomorphic computer comprising,a zoomormphic shellhaving a recognizable zoomorphic shape, a transceiver attached to thezoomorphic shell for communicating with the computer system, a positiondetecting unit for determining position of the zoomorphic shell relativeto the computer system, with change of position of the zoomorphic shellrelative to the computer system changing state of at least one of thezoomorphic shell and the computer system, a movable element attached tothe zoomorphic shell, and a feedback unit supported by the zoomorphicshell and maintainable in communication with the computer system, withthe feedback unit modifying a position of a movable element in responseto computer system output.
 19. The zoomorphic computer of claim 18,wherein the position of the movable element is modified in response toreceipt of electronic messages.